Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27406714195
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27406714195

@functionstackx could you please review this?

yes. lgtm once passes and then i will do /reuse on it https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27475314602/job/81213395277?pr=1715

hi @1am9trash it seems like conc

hi @1am9trash it seems like conc512 is failing, can u take a look? (will cancel the rest of the conc for now to avoid clogging up the CI queue since conc512 failed already)

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27475314602
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27475314602

Hi, @functionstackx
I checked the server logs for the failed task (conc=512) and found that it never actually started running successfully because it hit an Out of Memory (OOM) issue. This has never happened in our previous tests, even with higher concurrency levels.

Callback: Queue 0x796394800000 Aborting with error : HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. 
This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events.
Code: 0x1008 Available Free mem : 0 MB

My assumption is that there were other workloads on the server competing for resources at the time, and therefore the failure is unrelated to the v4 testing changes introduced in this PR. I reran the task (conc=512), and it completed successfully without encountering the issue.
Reran: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27475314602/job/81251602351?pr=1715

Thanks.